Employing labor-supply theory to measure the reward value of electrical brain stimulation

A model drawn from labor-supply theory is shown to provide a good account of time-allocation decisions taken by rats working for rewarding brain stimulation. The model makes it possible to infer, from behavioral data, the growth of the rewarding effect as a function of stimulation strength. Measurement of this function provides information about the stage of the reward circuitry where drugs or lesions alter the rewarding effect. The labor-supply model is used to illustrate how approaches drawn from economics, psychology, and neuroscience can inform each other. The model is linked to a set of psychological processes, including those responsible for transformation of the transient neural signal produced by the rewarding stimulation into an enduring record of payoff, estimation of a mean effort price, delay discounting, and estimation of the substitutability of work and leisure goods. All of these processes seem germane to economic behavior.  2004 Elsevier Inc. All rights reserved. 1. Neurobehavioral economics and the linkage of structure and function Behavioral economics bridges the economic and psychological study of human behavior. The aim is to ground economic theory in a realistic account of how people act in economic contexts. A mechanistic approach to achieving such an account involves spec* Corresponding author. E-mail address: [email protected] (P. Shizgal). 0899-8256/$ – see front matter  2004 Elsevier Inc. All rights reserved. doi:10.1016/j.geb.2004.08.003 284 K.L. Conover, P. Shizgal / Games and Economic Behavior 52 (2005) 283–304 ifying the psychological processes that are brought to bear on allocation decisions and determining their operating principles and interaction. Among the psychological processes that figure prominently in behavioral economic accounts are emotion, motivation, reward, perception, memory, evaluation, and decision making. It is a natural step to extend the behavioral economic approach to the neural underpinnings of the psychological processes of interest. In effect, such ventures into neurobehavioral economics represent a wager on the value of reverse engineering, the art of inferring functional principles from an analysis of brain structure. Evolution has shaped a computational device, the brain, which is vastly more sophisticated and powerful than any machine humans have yet devised. Behavioral and cognitive neuroscientists seek to gain insight into the design principles embodied in this device by examining the operation of circuits, cells, and molecules, both in the normal and damaged brain. Conversely, a well-elaborated description of a psychological process is essential in order for neuroscientists to figure out which bits of brain contribute to it and how. Thus, the study of structure and function inform each other and are mutually dependent. Establishing the mapping between structure and function is a subtle and challenging endeavor. In general, convergent findings from at least four different technical approaches are essential: lesions, recording, stimulation, and simulation. Lesion challenges can establish whether the integrity of a structure is necessary to the normal operation of a psychological process. This is an essential step, but is not sufficient in itself to link structure and function. There are many different ways in which neural computations could be altered by the removal of tissue or disruption of its operation, and not all of these necessarily operate in the intact brain. Analogous logic pertains to pharmacological challenges and genetic manipulations, such as knock-outs or reduction of transcription through administration of antisense DNA. Recording methods (which include electrophysiology, electrochemistry, and functional neuroimaging) can determine whether particular neural tissues are indeed engaged during operation of a psychological process and whether the observed variation of activity observed at the neural level corresponds, when suitably transformed, to the variation measured at the behavioral level. Nonetheless, recording methods are correlational and hence, they cannot, in isolation, distinguish the neural machinery responsible for a given process from co-activated circuitry that may subserve other functions. To establish causal relationships between structure and function, it is necessary to inject a signal into the neural circuitry in question, for example, by means of electrical stimulation, and to show that the artificially induced activity mimics the effects of natural stimuli or psychological states. Finally, computational modeling is required to determine whether the observed and injected neural signals and their sequelae can indeed account for the process they are purported to explain. The work presented in this paper adopts the third and fourth approaches to the linkage problem, stimulation and simulation, to address the neural mechanisms that compute payoffs and allocate behavior to harvest them. The experimental subjects are laboratory rats, and the model system is the neural circuitry responsible for the powerfully rewarding effect produced by electrical stimulation of the medial forebrain bundle. We propose that application of an economic model drawn from labor-supply theory allows us to infer, from behavioral data, the magnitude of a neural signal implicated in the computation of payoffs. K.L. Conover, P. Shizgal / Games and Economic Behavior 52 (2005) 283–304 285 We propose further that application of the model provides information about the stage(s) of processing where drugs and lesions alter the computational machinery. Some use of electrophysiological recordings has already been made in the study of brain stimulation reward. We suggest below that to realize the full power of recording methods, the economic models of the kind employed here must be translated into real-time models that are psychologically realistic. Some speculations about the nature of such a real-time model are provided, and we discuss how the identification of the neural circuitry responsible for the rewarding effect of electrical brain stimulation could have implications for understanding economic behavior. 2. Brain stimulation reward Electrical brain stimulation is deemed to be rewarding when the subject is willing to work to obtain it. Such rewarding effects have been demonstrated across the vertebrate phylum, from goldfish to humans (Bishop et al., 1963; Boyd and Gardiner, 1962; Distel, 1978; Lilly and Miller, 1962; Olds and Milner, 1954; Porter et al., 1959; Roberts, 1958). Figure 1, a summary of brain stimulation reward sites in the rat (Wise, 1996), illustrates their widespread distribution. The electrodes employed in the study described below were aimed at the lateral hypothalamic level of the medial forebrain bundle (MFB), a “multi-lane expressway” linking many forebrain structures to midbrain and hindbrain structures. The behavior of a rat working for maximally rewarding stimulation of the MFB is striking to behold. To obtain such stimulation, rats will press levers for hours on end, cross electrified grids, leap over hurdles while racing uphill, or forgo eating despite severe food deprivation. There is good evidence that the neural signals responsible for this potent reFig. 1. A sagittal sketch of a rat brain showing regions (dashed ellipses) where electrical brain stimulation has been shown to produce rewarding effects (from Wise, 1996). The long cross-hatched region represents the medial forebrain bundle, which passes through the lateral hypothalamus and ventral tegmental area. Other effective sites are designated by dashed ellipses. 286 K.L. Conover, P. Shizgal / Games and Economic Behavior 52 (2005) 283–304 ward arise in neurons that process naturally occurring stimuli. For example, the rewarding effect of MFB stimulation can compete with (Conover and Shizgal, 1994), summate with (Conover and Shizgal, 1994; Conover et al., 1994), and substitute for (Green and Rachlin, 1991) the rewarding effects of gustatory stimuli such as sucrose solutions, rat chow, and water. Opioids and psychomotor stimulants, enhance the rewarding effect of MFB stimulation (Wise, 1996), lending indirect support for the view that brain circuitry involved in brain stimulation reward contributes to drug-seeking behavior (Wise, 1996). Several key features of the circuitry responsible for the rewarding effect of MFB stimulation are illustrated in Fig. 2. The stimulation consists of a series (“train”) of current pulses. Each pulse elicits a volley of action potentials (nerve impulses) in fibers passing near the tip. The post-synaptic effect of the series of volleys is integrated by the circuitry represented by the symbol. This circuit appears to be arranged so as to implement an aggregate rate code (Gallistel et al., 1981; Shizgal, 1999; Shizgal and Conover, 1996). Signal strength in such a code reflects the total number of action potential elicited within a given time window; it matters not whether this total is achieved by firing a few fibers at a high rate or many fibers at a low rate. The curve plotted in the center of Fig. 2 represents the “reward-growth” function that translates the firings aggregated over the stimulation train into the intensity of the rewarding effect (Gallistel and Leon, 1991). The principal subject of this paper is the allocation mechanism (represented by the question mark) that translates the output of the reward-growth function into behavior. The experimenter controls the electrical stimulation, sets the conditions for its delivery, and observes the behavioral output (e.g., lever pressing). In order to measure the rewardgrowth function, it is necessary to account for the allocation mechanism that intervenes between the output of this function and the observable performance of the rat. Before turning to consideration of the allocation function and the application of economic ideas to its derivation and measurement, let us first consider why the reward-growth function is of interest. Fig. 2. A minimal model of signal processing in the brain circuitry responsible for intracranial self-stimulation. The stream of action potentials triggered by the stimulation is integrated over time and space and transformed into a signal representing the intensity of the reward. The activity evoked by the stimulation is interpreted according to an “aggregate rate code”; reward intensity depends only on the total number of firings within a given time window and not on their spatial or temporal distribution. The function that translates the aggregate firing rate (or the stimulation variables that determine this rate) into reward intensity is called the “reward-growth function.” A second function, which translates reward intensity into behavior (e.g., lever-pressing, alley-running, etc.), is called the “allocation function.” K.L. Conover, P. Shizgal / Games and Economic Behavior 52 (2005) 283–304 287 Fig. 3. Different ways in which manipulations such as lesions or drugs could alter the reward-growth function (A). Variables acting prior to the input shift the reward-growth function laterally (B), whereas variable action beyond the output shift the function vertically (C). A simulated reward-growth function (Eq. (5), below) is shown in Fig. 3a. The abscissa gives the number of stimulation pulses per fixed-duration train. In this example, the reward effect saturates once the number of pulses exceeds one hundred. Knowing the saturation point would be extremely useful to an electrophysiologist searching for neurons in the portion of the circuit beyond the stage that computes the reward-growth function. One indication that a population of neurons may be carrying the reward signal would be invariance of stimulation-induced activity when the number of stimulation pulses exceeds one hundred. Similarly the slope of the rising portion of the curve provides another criterion for determining whether a population of neurons may be carrying the reward signal. Another important use of the reward-growth function is to determine at which stage of the reward circuitry a manipulation such as a lesion or a drug produces its effect. Figure 3b illustrates an effect of a manipulation acting prior to the input to the reward-growth function. By increasing or decreasing the impact of each stimulation pulse, the manipulation in question shifts the reward-growth function along the abscissa. Figure 3c illustrates the effect of a manipulation acting at a stage of the circuitry subsequent to the output of the reward-growth function. By rescaling the output of the function, the manipulation in question shifts the curves along the ordinate. Thus, if we can determine 288 K.L. Conover, P. Shizgal / Games and Economic Behavior 52 (2005) 283–304 the reward-growth function, we can narrow down the stage of the circuitry where lesions and drugs alter the rewarding impact of the stimulation. The input to the reward-growth function is under the control of the experimenter, but the output is not directly observable. In order to measure the output, a well-validated model of the behavioral allocation function is required. We have developed such a model on the basis of labor-supply theory. 3. Application of labor-supply theory to the study of BSR The theory of labor supply (Frank, 1999) provides a model of how people determine the balance between work and leisure. Both classes of activity are worthwhile, but the rewards derived from work (e.g., money, status, recognition of achievements) are different in kind from the rewards derived from leisure (e.g., pursuit of family and personal interests). In economic terms, rewards from work are only partially substitutable for rewards from leisure. We will introduce the concept of substitutability in the more familiar context of consumer choice and then extend the concept to labor supply. In the context of consumer choice, substitutability refers to the sensitivity of allocation decisions to changes in price (once the effect of price changes on purchasing power has been factored out). A red pen and blue pen are perfect substitutes for a consumer who is indifferent to ink color. Thus, when the prices of the two types of pens differ, the consumer buys only the cheaper variety. In contrast, right and left shoes are perfect complements. Even if a shoe shop were to charge different prices for right and left shoes, the consumer would nonetheless buy the same quantity of each type. The effect of price changes on purchasing decisions concerning imperfect substitutes falls between the effects on decisions concerning perfect substitutes and perfect complements. Pencils and pens are imperfect (partial) substitutes; either can serve for some writing needs, but one or the other is required if erasure or permanence is called for. Changes in the relative prices of pencils and pens will alter the relative quantities purchased, but the consumer will try to include some of each in the mix. In the context of labor supply, substitutability can be conceived as the sensitivity of time-allocation decisions to changes in wages. If rewards from work were perfectly substitutable for rewards from leisure, then a utility maximizer would either work all the time or not at all, depending on which rewards were higher. If rewards from work and leisure were perfect complements, then the division of time between work and leisure would be fixed and would not change as a function of wages. In the intermediate case of imperfect substitutability, changes in allocation in response to a change in wages would depend on the current allocation of time between work and leisure. A modest increase in wages would be sufficient to lead an underemployed person to allocate considerably more time to work. However, even a substantial increase in wages would be insufficient to lead a person holding down two full-time jobs to allocate much additional time to work. Mathematical models of how wage changes affect work and leisure time can be derived from functions that compute the utility of different combinations of work and leisure under the assumption that employees maximize utility. What follows is a demonstration that such a model also provides a good description of operant behavior in rats and provides a theoretical basis for measuring the reward value of electrical brain stimulation. The experiments K.L. Conover, P. Shizgal / Games and Economic Behavior 52 (2005) 283–304 289 were carried out in rats working for stimulation of the medial forebrain bundle at the level of the lateral hypothalamus. The surgical preparation, apparatus, and training procedures were similar to those employed in earlier work (Conover et al., 2001). Experiments in which animals must perform a task in order to obtain reward have long been interpreted in economic terms (Allison, 1983; Kagel et al., 1995; Rachlin et al., 1976; Staddon, 1987). Consider, for example, a typical experiment in which a rat can press a lever to obtain rewarding brain stimulation. In economic terms, time at the lever is classified as work, and time away from the lever is classified as leisure. In congruence with the economic analysis of human behavior, work goods, such as food, water and brain stimulation, have been shown to be only partially substitutable in laboratory animals for leisure goods, such as rest, exploration and grooming. The labor-supply model that we used to scale brain stimulation reward is based upon a constant elasticity of substitution utility function (Arrow et al., 1962). Similar functions have been used in previous work on brain stimulation reward and also with natural rewards such as food and water (Green and Rachlin, 1991; Rachlin, 1982). The form of the equation used here is: u= (vl × l + vb × bs)1/s (1) where u is the utility, l the time spent in leisure, b the number of rewards, vb the reward value of the brain stimulation, vl value of a unit of leisure, and s the coefficient of substitutability between leisure goods and brain stimulation goods, a dimensionless number that may vary from 1 for perfect substitutes, to minus infinity for perfect complements. To determine whether the labor-supply model provides a good description of the behavior of rats working for rewarding brain stimulation, we arranged the experimental contingencies to ensure that the rats had to give up an experimenter-controlled amount of leisure time in order to harvest each reward. To obtain a stimulation train, the rat had to be holding the lever down at the end of an interval drawn from an exponential distribution; if the rat failed to keep the lever depressed at the end of the interval, then no reward was delivered. This “free-running variable-interval” schedule of reinforcement pays the rat in proportion to the amount of time worked. The rat’s budget is simply the time that the stimulation is made available, and the rat is free to allocate this budget between work and leisure as it sees fit. Unlike the case of the traditional variable-interval schedule used widely in operant conditioning studies (Rachlin, 1989), the budget constraint is linear: